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Abstract 

This  paper  will  describe  the  development  of  a  protocol  for  probabilistic  reliability  assessment  for  SHM  systems 
as  well  as  present  an  experimental  demonstration  for  a  vibration-based  structural  damage  sensing  system.  The 
results  of  the  full  validation  study  highlight  the  general  protocol  feasibility,  emphasize  the  importance  of 
evaluating  key  application  characteristics  prior  to  the  POD  study,  and  demonstrate  an  approach  to  quantify 
varying  sensor  durability  on  the  POD  performance.  Challenges  remain  to  properly  address  long  time -scale 
effects  with  accelerated  testing  and  large  testing  requirements  due  to  the  independence  of  the  inspection  of  each 
flaw  location. 

Keywords:  Model-assisted  POD  evaluation,  probability  of  detection  (POD),  reliability,  structural  health 
monitoring 


1.  Introduction 

The  successful  deployment  of  systems  for  health  monitoring  of  structures  depends  on 
appropriate  verification  and  validation  (V&V)  of  these  structural  health  monitoring  (SHM) 
systems.  The  V&V  method  must  explicitly  evaluate  all  aspects  of  the  SHM  system  that  can 
affect  its  capability  to  detect,  localize,  or  characterize  damage.  Moreover,  it  must  evaluate  the 
effects  that  usage  and  environmental  conditions  have  on  these  capabilities  over  time.  The 
current  U.S.  Air  Force  practice  for  maintaining  aircraft  structures  follows  the  Aircraft 
Structural  Integrity  Program  (ASIP)  methods,  as  documented  in  MIL-STD-1530C  [1].  A 
critical  part  of  the  damage  tolerance  approach  is  the  assessment  of  the  reliability  of 
Nondestructive  Evaluation  (NDE)  methods  that  are  used  for  the  periodic  inspection  of 
structures.  MIL-HDBK-1823A  provides  guidance  on  probabilistic  methods  for  NDE 
reliability  assessment  and  introduces  the  use  of  models  to  complement  experimentation  for  a 
probability  of  detection  (POD)  determination  [2],  As  SHM  methods  depending  on  permanent, 
on-board  mounted  damage  sensing  systems  continue  to  be  proposed  and  developed  for 
complementing  ground-based  NDE  inspections  for  aircraft  structural  integrity  purposes,  it  is 
necessary  that  the  reliability  of  these  damage  sensing  systems  be  assessed  with  a  rigor  that  is 
suitable  and  sufficient  for  the  function  that  they  are  expected  to  perform  within  the  ASIP 
methodology  [3].  For  damage  detection,  this  necessarily  results  in  the  need  for  a  POD 
determination.  For  localization  and  characterization,  the  metrics  and  their  evaluation  process 
are  the  subject  of  current  research  and  development  [4].  This  paper  will  present  a 
demonstration  of  the  damage  detection  protocol  utilizing  empirical  data,  models,  and 
uncertainty  analyses  for  characterizing  SHM  reliability  [5]. 

2.  Protocol 

An  outline  of  the  MAPRA  protocol  is  given  in  Figure  1.  The  MIL  HDBK  1 823 A  for  USAF 
NDE  certification  based  on  POD  is  the  foundation  of  the  protocol  [2].  In  addition,  model- 
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assisted  approaches  must  be  applied  to  address  limits  of  experimentation,  facilitate  proper 
uncertainty  analysis  for  damage  detection  cases  and  expand  the  assessment  to  quality  of 
localization/characterization  estimates.  This  protocol  includes  four  critical  components:  (1)  a 
procedure  to  identify  the  critical  factors  impacting  SHM  system  performance;  (2)  a  multistage 
or  hierarchical  approach  to  SHM  system  validation;  (3)  a  model-assisted  evaluation  process  to 
address  the  wide  range  of  expected  damage  conditions  that  cannot  be  experimentally  tested; 
and  (4)  POD,  probability  of  false  call  (POFC)  and  probability  of  random  missed  call  (POMC) 
evaluations  with  confidence  bounds  estimation  and  uncertainty  analysis  for  damage  detection 
SHM  systems,  and  evaluation  of  appropriate  probabilistic  metrics  to  characterize  the  quality 
of  damage  localization  and  damage  characterization  for  SHM  systems  that  include  such 
capabilities.  The  multistage  validation  approach  is  designed  to  incrementally  test  SHM 
systems  with  structures  of  increasing  complexity.  The  multistage  approach  includes  (a) 
laboratory  testing  of  relevant  flaws,  (b)  laboratory  sub-component  testing  including 
environmental  and  loading  conditions,  (c)  a  system  level  life-testing  (full-scale  fatigue  testing 
if  feasible),  (d)  on-structure  demonstration,  and  (e)  final  system  verification. 


Figure  1.  Outline  of  protocol  for  SHM  validation. 


A  model-assisted  strategy  for  the  design  and  execution  of  POD  studies  for  NDE  has  been 
developed  and  demonstrated  to  help  mitigate  validation  costs  and  to  improve  POD  evaluation 
quality  by  addressing  a  wider  array  of  inspection  variables.  By  including  greater 
sophistication  in  the  models,  there  should  be  less  error  present  between  the  model  and 
available  experimental  data.  By  addressing  variability  in  the  model  and  minimizing 
unexplained  error  in  the  representation,  less  experimental  data  will  be  required  to  address  the 
unknowns  in  the  evaluation.  The  following  opportunities  in  the  POD  model  evaluation  and 
application  steps  have  the  potential  to  impact  sample  and  testing  requirements:  (a)  careful 
model  factor  selection  addressing  system  variation,  (b)  physics-based  model  calibration 
including  uncertainty  bounds  assessment  for  the  specific  inspections  of  interest,  (c)  controlled 
physics-based  model  validation  to  ensure  the  model  is  valid  over  desired  range  of  application, 
(d)  evaluation  of  POD  using  two-level  analysis  to  address  input  parameter  variability  with 
uncertainty  bounds,  (e)  integration  of  experimental  data  generated  from  a  designed 
experiment  using  a  Bayesian  framework  to  revise  the  prior  distributions  of  inputs  and  achieve 
new  posterior  distributions,  and  (f)  inverse  methods  to  ideally  address  all  uncontrolled 
parameter  variations  in  the  measurement.  More  information  on  the  protocol  can  be  found  in 
reference  [5]. 

3.  Experimental  Study  Setup 

The  example  used  for  this  initial  demonstration  of  the  protocol  is  a  system  for  detecting  the 
presence  of  damage  using  permanently  mounted  transducers.  A  test  article  representing  an 
aircraft  structure  of  medium  complexity  was  designed  and  built.  The  test  article  consists  of 
three  plates  connected  by  two  lap  joints  with  fasteners  (see  Figure  2).  In  addition,  a  fixture 
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was  built  for  supporting  the  test  artiele.  Fatigue  eraek  damage  around  the  fastener  holes  ean 
be  simulated  by  manually  ereated  thin  euts  at  seleeted  loeations.  The  test  fixture  design 
provides  the  eapability  to  vary  critical  parameters  of  the  system  with  a  focus  on  force  loading 
boundary  conditions,  joint  fastener  torque  conditions,  and  temperature. 

The  initial  demonstration  on  this  test  article  and  fixture  uses  a  vibration  based  damage 
detection  method.  An  ETrema  brand  Terfenol-D  magnetostrictive  actuator  was  used  for  band- 
limited  pseudo-random  excitation  up  to  1200  Hz,  and  the  dynamic  response  of  the  plate  was 
recorded  using  eight  50mV/g  single-axis  accelerometers  set  to  measure  out-of-plane  motion 
and  provide  input  to  change  detection  algorithms.  The  accelerometers  were  placed  at  locations 
that  attempt  to  maximize  the  ability  to  detect  change  in  the  modal  dynamics  of  the  structure 
even  in  the  presence  of  modeling  errors.  M-Bond  200  adhesive  was  used  for  semi¬ 
permanently  affixing  the  accelerometers  to  the  bottom  of  the  assembly.  Four  bonded  foil  350- 
ohm  strain  gages  were  also  installed  on  the  test  article  for  tracking  the  state  of  stress  under 
mechanical  loading.  Finally,  thermocouples  were  also  used  for  studying  the  effects  of  heating 
due  to  excitation  of  the  damage  detection  system  and  the  effects  of  the  operational 
temperature.  The  layout  of  the  sensors  is  presented  in  Figure  2.  A  Lab  VIEW  data  acquisition 
system  was  used  for  acquiring  the  required  data.  Variations  in  operational  temperature  were 
simulated  by  testing  the  system  inside  a  carefully  controlled  Thermotron  SE-1200 
environmental  chamber. 

The  method  for  inducing  damage  involved  notching  the  area  of  the  „skin''under  a  fastener.  To 
initially  isolate  damage  to  the  skin  alone,  the  test  article  design  includes  machined  relief  slots 
in  the  joining  plate  at  the  fastener  locations.  A  change  metric  based  on  the  area  under  the 
curve  representing  the  difference  between  two  frequency  responses  from  200  to  1200  Hz  and 
an  R-square  metric  were  used  for  assessing  changes  in  structural  dynamics  due  to  mechanical 
and  thermal  loading,  actual  damage,  and  combinations  thereof,  and  therefore  attempting  to 
detect  damage  presence  and  growth. 


sensor  key 

o 

□ 

A 

(in  air  on  top) 


accelerometer 


thermocouple 


strain  gauge 


Figure  2.  Location  of  sensors  on  the  test  article. 


3 

Approved  for  public  release;  distribution  unlimited. 


4.  Factor  Evaluation  Studies 


Following  the  protocol,  prior  to  designing  the  validation  study  test  matrix,  the  following 
factors  were  assessed  through  controlled  studies:  (a)  mass  loading  and  unloading,  (b)  fastener 
torque,  (c)  boundary  condition  variation,  (d)  temperature  variation  and  temperature  gradients, 
(e)  sensor  bond  quality,  (f)  ambient  noise,  and  (g)  flaw  growth.  More  details  on  the  factor 
evaluation  studies  are  presented  in  reference  [6]. 

Thermal  loading  studies  were  performed  by  varying  the  ambient  temperature  from  -20°F  to 
150°F.  During  this  study,  the  thermal  capacity  of  the  panel  end  conditions  fixtures  was  found 
to  produce  significant  thermal  gradients  across  the  test  article.  During  heating  and  cooling 
periods,  temperature  gradients  as  high  as  45°F  across  the  test  specimen  were  observed.  For 
validation  studies,  an  estimate  of  expected  gradients  „in  the  field”  is  needed.  An  assumption 
was  made  for  the  validation  study  that  temperature  gradients  in  the  region  of  interest  will  be 
primarily  limited  to  +/-  10°F. 

To  address  the  temperature  conditions,  a  new  temperature  compensation  algorithm  was 
designed  and  implemented  that  (1)  uses  multiple  FRF  references  at  different  temperature 
bands  and  (2)  incorporates  a  nonlinear  frequency  shift  model  for  local  temperature 
compensation.  An  approach  was  tested  that  uses  three  baselines,  one  at  the  nominal 
temperature,  one  at  a  higher  temperature  range  (with  minimal  temperature  gradients)  and  one 
at  the  low  temperature  range.  The  algorithm  then  evaluates  the  minimum  of  the  three  damage 
metric  results.  A  two  parameter  frequency  shift  model  was  introduced 


fnewif)  -f  + 


,1000  Hz 


jf  +  ^0’ 


(1) 


where  ^  and  represent  bias  and  slope  adjustments,  respectively.  A  nonlinear  least  squares 
estimation  routine  was  implemented  to  evaluate  (/>o  and  and  a  log  transform  of  the  FRFs 
was  used  to  enable  sensitivity  to  changes  all  along  the  FRF,  not  just  at  the  peaks.  Through  the 
use  of  this  temperature  compensation  algorithm,  significant  improvements  to  the  damage 
metric  were  achieved.  One  downside  with  the  algorithm  is  that  the  fit  and  interpolation 
process  take  longer  (seconds)  to  run.  As  well,  there  is  still  some  difficulty  in  compensating  for 
rapid  temperature  changes  that  produce  severe  gradients  across  the  plate. 

Failure  of  accelerometer  bonding  was  observed  several  times  during  factor  thermal  testing. 
These  failures  occurred  during  the  prolonged  high  temperature  runs  at  150°F.  Coherence  was 
shown  to  be  a  viable  metric  to  monitor  bond  quality  over  time.  While  vibration-based  damage 
detection  systems  are  proposed  as  global  methods  with  some  sensor  redundancy,  relying  on  a 
single  reference  sensor  will  result,  upon  bond  degradation,  in  either  highly  degraded 
performance  and/or  complete  failure  of  the  damage  detection  system  well  before  the  structure 
end-of-life  is  reached.  Sensor  and  sensor  bonding  reliability  must  therefore  be  accurately 
assessed  as  part  of  a  validation  study. 

The  noise  generated  by  the  environmental  chamber  and  by  other  equipment  in  the  laboratory 
posed  an  opportunity  for  studying  the  effects  of  ambient  noise  on  the  detection  capability. 
Coherence  levels  were  observed  to  change  depending  on  whether  the  unit  was  cooling  or 
heating  and  the  „thiDttle"  level  of  the  chamber,  and  on  whether  the  chamber  door  was  open  or 
closed.  For  the  validation  study,  both  „diamber  on"  and  „diamber  off'  conditions  will  be 
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acquired  in  order  to  inelude  in  the  analysis  the  effects  of  varying  noise  eonditions  that  are 
expected  in  the  field. 

An  initial  study  on  the  effect  of  damage  growth  was  performed  to  ensure  adequate  sensitivity 
during  the  final  validation  study.  Customized  XAeto^'^  blades  were  used  to  make  cuts  in  the 
aluminum  plates.  The  resulting  notch  width  was  found  to  be  0.012"  +/-  0.002".  Cuts  were 
initially  made  at  0.063"  (1/16")  inerements  up  to  0.63".  For  the  first  series  of  outs  up  to  0.25", 
sensitivity  to  notoh  length  inoreases  was  observed,  but  the  trend  was  small  relative  to  noise, 
and  not  quite  linear.  Greater  sensitivity  to  the  larger  cuts  was  observed  and  olear  sensitivity  to 
notohes  on  the  order  of  0.63"  was  demonstrated.  Note,  a  signifioant  inorease  in  the  damage 
metrio  was  observed  after  a  two  week  delay  between  the  end  of  the  0.25"  notoh  out  and  the 
start  of  the  0.31"  notoh  out.  Relaxation  of  the  boundary  conditions  over  time  was  thought  to 
be  the  souroe  of  the  ohange.  For  validation,  oontrolled  time  delays  should  be  included  into 
such  studies  to  isolate  and  addre 

5.  Design  of  Validation  Study 

The  validation  study  oonsisted  of  growing  flaws  by  artifioially  cutting  the  struoture  at  two 
fastener  site  looations,  site  2  and  site  3,  as  shown  in  Figure  2.  A  series  of  environmental  and 
boundary  conditions  were  studied  after  eaeh  flaw  growth  seenario;  temperature  variation  (+/- 
40°F),  temperature  gradients,  loading  and  unloading  of  10  lb  mass,  a  simulated  maintenance 
aetion  at  a  set  of  fasteners  (see  Figure  2)  ineluding  the  ease  of  minor  loosening,  and 
reinstallation  and  replaeement  of  aecelerometers.  During  the  flaw  growth  period,  tests  were 
performed  before  and  after  flaw  growth,  and  after  fastener  installation.  Temperature  chamber 
states  of  „on"and  „off'with  the  ambient  temperature  set  to  T  =  72°F  were  aequired  for  all  test 
eonditions.  Five  averages  were  taken  for  all  SHM  system  aequisitions.  After  any 
environmental  eondition  or  change  to  the  test  fixture,  testing  was  performed  with  the  ambient 
temperature  returned  to  T  =  72°F. 

6.  POD  Analysis  Approach 

A  summary  deseription  of  the  approaeh  being  followed  for  obtaining  a  probability  of 
detection  model  for  on-board  damage  deteetion  cases  follows.  Conventional  probability  of 
deteetion  (POD)  evaluation  for  many  quantitative  NDE  applications  first  uses  empirieal  data 
to  evaluate  statistieal  relationships  between  the  measurement  response,  a ,  and  the  primary 
flaw  size  variable,  a.  Through  applieation  of  a  deteetion  eriterion  as  part  of  the  NDE 
proeedure,  this  statistieal  „a  versus  a  ’  model  can  be  used  for  evaluating  the  POD  curve  and 
probability  of  false  eall  (POEC)  rate,  whieh  together  are  usually  referred  to  as  “a  POD 
model”.  The  deteetion  system  can  be  abstraetly  represented  by  a  set  of  random  variables  a, 

that  act  as  inputs  to  a  measurement  model.  Input  variables  ean  be  categorized  as  being 
eontrolled  (e.g.  flaw  size  and  material  properties)  or  uneontrolled  (e.g.  liftoff,  flaw 
morphology,  and  measurement  noise).  Deteetion  eonsists  of  the  measurement  model  output  d 
being  elassified  (or  “ealled”)  aeeording  to  pre-specified  rules  (e.g.  a  threshold). 

The  model-assisted  POD  (MAPOD)  approaeh  proposes  to  replaee  a  eonventional  statistieal  fit 
in  the  measurement  model  with  a  eomplete  physics-based  model,  f,  ealibrated  for  a  given  set 
of  experimental  eonditions.  This  relationship  is  given  by 

a  =  +  (2) 
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where  y^o  and  fi\  represent  the  model  calibration  parameters,  and  ^  represents  the  residual  error 
between  the  model  and  the  experimental  data.  Estimating  the  statistics  of  y^o,  Pu  and  s 
necessitates  specific  experimental  sampling  requirements.  Variations  due  to  flaw  size  and 
environmental  (noise)  conditions,  for  example,  are  represented  in  the  model  as  probability 
distributions  of  the  input  variables.  Hybrid  models  incorporating  both  empirical  and  physics- 
based  components  can  be  implemented  to  address  all  key  factors  including  those  that  cannot 
be  adequately  simulated.  For  this  study,  due  to  a  lack  of  a  validated  physics-based  model,  a 
surrogate  model  fit  using  empirical  data  was  applied  in  the  evaluation. 

In  this  study,  the  primary  variable  associated  with  the  critical  flaw  size  is  crack  (notch)  length, 
a\.  Controlled  secondary  variables  in  the  study  include  flaw  location  (ai),  mean  temperature 
(as),  temperature  gradients  (a4),  ambient  noise  level  (as).  A  response  surface  methodology 
was  applied  here  to  estimate  the  effect  of  each  factor  on  the  damage  metric  response  and 
construct  a  model, /(a J  including  uncertainty.  Random  events  such  as  sensor  failure  / 
disbond  (hi),  sensor  bond  degradation  (hi),  sensor  replacement  (hs),  and  local  maintenance 
actions  (^4)  were  considered  in  the  POD  evaluation  study.  Assumptions  concerning  their 
frequency  can  be  made  and  empirical  models  representing  their  effect  can  be  evaluated  and 
applied  in  conjunction  with  the  scope  of  the  SHM  application. 

To  complete  the  POD  evaluation,  an  assessment  of  the  detection  model  under  varying  input 
conditions  including  uncertainty  propagation  is  necessary.  In  [10],  a  second-order 
probabilistic  approach  has  been  developed  to  propagate  both  aleatory  uncertainty,  due  to 
inherent  randomness  in  system  behavior,  and  epistemic  uncertainty,  due  to  a  lack  of 
knowledge  about  values  expected  to  be  fixed.  Using  this  approach,  epistemic  variables  are 
specified  as  intervals  on  values  of  parameters  such  as  the  means  and  standard  deviations  of 
random  variables.  Monte  Carlo  analysis  is  applied  here  using  outer  and  inner  loops.  The  outer 
loop  varies  the  values  of  distribution  parameters  of  selected  epistemic  variables  while  the 
inner  loop  samples  from  the  distributions.  For  this  study,  the  distributions  for  the  input 
variables,  mean  temperature  and  temperature  gradients,  are  presented  in  Figure  3. 
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Figure  3.  Distributions  for  temperature  parameters:  (a)  mean  normalized  temperature  (7),  (b) 
maximum  temperature  gradient  (dTmax  /  10°F). 


6 

Approved  for  public  release;  distribution  unlimited. 


7.  POD  Evaluation  Results 


7.1  Sensitivity  of  POD  to  Flaw  Location 

Following  acquisition  of  the  experimental  data,  a  regression  model  fit  was  performed  to 
evaluate  /(a,)  using  the  R  software  environment.  Three  different  flaw  models  were 
considered  in  the  evaluation:  (a)  a  flaw  2  and  3  combined  evaluation  (including  flaw  location 
factor,  afj,  (b)  a  flaw  2  evaluation  only,  and  (c)  a  flaw  3  evaluation  only.  One  reason  for 
performing  and  studying  separate  model  fits  for  the  different  flaw  growth  sites  was  due  to 
early  observations  that  the  SHM  system  was  more  sensitive  to  changes  in  flaw  3  with  respect 
to  changes  in  flaw  2.  POD  analysis  results  for  the  vibration-based  SHM  study  are  presented  in 
Figure  4  with  respect  to  flaw  size  (in  inches)  for  the  case  of  a  damage  call  threshold  of  0.05. 
The  damage  metric  initially  tested  was  based  on  taking  the  median  response  from  all  of  the 
active  frequency  response  functions  (FRF),  in  order  to  minimize  sensitivity  to  outlier  sensor 
responses  during  operation.  For  each  POD  evaluation,  both  input  parameter  variation  and 
model  uncertainty  are  addressed  through  a  two-level  Monte  Carlo  simulation.  To  minimize 
simulation  time,  2000  samples  were  used  for  eaeh  level,  and  the  POD  evaluation  was 
performed  at  51  different  flaw  sizes  from  0.0  to  1.0  inch. 

From  these  results,  there  is  clear  need  to  separately  evaluate  the  POD  models  for  flaw  2  and 
flaw  3  locations.  A  single  POD  curve  does  not  properly  address  the  poor  detection  capability 
at  the  flaw  2  loeation  as  a  function  of  performance  at  large  flaw  size.  Using  the  ‘flaw  2  and  3 
combined’  results  will  give  one  a  false  sense  of  security  in  terms  of  detection  capability.  Note, 
for  any  future  SHM  validation  study,  care  must  be  taken  to  ensure  the  ‘overall’  POD 
capability  evaluations  do  not  mask  ‘isolated’  flaw  locations  that  have  poor  detection 
capability.  Likewise,  the  low  false  call  rate  in  the  flaw  3  model  is  likely  due  to  the  flaw  3 
model  only  including  a  portion  of  the  simulation  study  variation.  In  future  work,  the  false  call 
model  should  try  to  include  all  experimental  study  data.  Lastly,  POD  results  for  flaw  2  shown 
in  Figure  4(b)  can  be  improved  by  using  only  the  optimal  sensor  #6  data. 


(in.)  (a)  a.|  (in.)  (b)  (in.)  (c) 

Figure  4.  POD  results  with  respect  to  flaw  size  including  uncertainty  bounds  for  (a)  combined  flaw  2 
and  3,  (b)  flaw  2  only,  and  (c)  flaw  3  only  {dmjhreshold  =  0.05,  use  median  sensor  response). 


7.2  Impact  of  Sensor  Degradation  over  Time 

This  analysis  approach  enables  the  evaluation  of  the  impact  of  sensor  durability  on  POD 
performance.  To  perform  this  study,  a  strategy  to  address  sensor  durability  issues  must  first  be 
clearly  defined  upfront,  as  shown  in  Figure  5.  Two  sensor  failure  seenarios  are  of  particular 
interest  here:  failure  of  the  optimal  detection  sensor  (accelerometer  6),  and  failure  of  the 
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reference  sensor.  For  the  example  of  „liaw  3"  results  shown  in  Figure  5,  it  is  clear  how  the 
corresponding  POD  curves  are  degraded  with  respect  to  the  optimum  case. 

Next,  models  for  sensor  performance  degradation  (i.e.  failure  rates)  are  required  for  the 
evaluation.  To  represent  the  failure  rate  of  in-situ  sensors  like  strain  gauges  or  accelerometers, 
a  ,Jbathtub''  curve  is  often  used  [8].  It  describes  a  particular  form  of  a  hazard  function  which 
comprises  three  parts:  an  initial  decreasing  failure  rate,  known  as  infant  failures,  a  second  part 
representing  a  constant  failure  rate,  known  as  random  failures,  and  a  final  part  representing  an 
increasing  failure  rate,  known  as  wear-out  or  fatigue  failures.  Evidence  from  strain  gauge 
sensor  data  on  C-17  aircraft  demonstrates  the  need  for  assessing  the  impact  of  degradation, 
where  22%  of  the  sensors  were  infant  failures  and  about  40%  of  the  total  failed  within  the  first 
ten  years  of  the  aircraft  life  [9].  Given  that  only  eight  sensors  are  present  in  the  subject  SHM 
system,  the  scenario  of  25%  failures,  two  accelerometers,  will  be  considered  during  the  first  6 
year  period  of  operation.  Figure  6(a)  presents  two  probability  density  functions  for  the  time  to 
failure  for  the  first  and  second  sensors.  Here,  gamma  distributions  are  applied  to  represent 
portions  of  the  ,bathtub''  curve. 

Data  tables  were  constructed  evaluating  POD  models  for  all  of  the  „sin^e  sensor”  and  'two 
sensor”  failure  scenarios.  A  Monte  Carlo  simulation  was  then  performed  using  10000  samples 
from  the  time  to  failure  distributions  for  the  first  and  second  sensors.  Equal  probability  of 
failure  was  assumed  for  the  eight  accelerometer  locations.  The  POD(t)  curve 
was  evaluated  at  0.1  year  time  increments  for  6  years.  Results  are  presented  in  Eigure  6(b)  for 
the  mean  value  from  the  composite  Monte  Carlo  simulation  POD  results  at  a  flaw  size  of  1 .0 
inch  as  a  function  of  time.  Results  for  both  flaw  2  and  flaw  3  locations  are  presented.  This 
analysis  is  useful  because  it  highlights  the  sensitivity  of  certain  flaw  locations  to  degradation 
in  the  SHM  system.  In  particular,  the  detection  of  flaw  2  suffers  from  weak  crack  sensitivity 
with  respect  to  significant  noise  sensitivity  due  to  varying  temperature  conditions.  Elaw  3 
detection  capability  was  found  to  be  quite  robust  to  the  failure  of  only  one  or  two 
accelerometers. 

8.  Summary 

This  work  has  presented  the  results  of  a  demonstration  featuring  the  application  of  a  proposed 
validation  protocol  to  a  vibration-based  structural  damage  sensing  system.  The  design  and 
results  of  the  full  validation  study  highlight  the  general  protocol  feasibility,  emphasize  the 
importance  of  evaluating  the  key  application  characteristics  prior  to  the  POD  study,  and 
demonstrate  an  approach  to  quantify  varying  sensor  durability  on  the  POD  performance. 
However,  challenges  remain,  in  particular  on  how  to  properly  address  long  time-scale  effects 
with  accelerated  testing  and  how  to  address  large  testing  requirements  given  the  independence 
of  each  flaw  location  in  the  evaluation.  Going  forward,  by  better  leveraging  validated 
numerical  models,  it  may  be  feasible  to  address  these  challenges. 
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Scenarios 
Addressing 
Sensor  Faiiure 
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Approach  2:  (Accei.#1  Faiiure) 
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Figure  5.  Sensor  failure  scenarios  with  matching  changes  in  POD  and  false  call  rate  ('flaw  3'  model). 


Figure  6.  (a)  Case  study  probability  density  functions  for  the  time  to  failure  for  the  first  sensor 
and  second  sensor,  (b)  mean  expected  probability  of  detection  (POD)  at  a  flaw  size  of  1 .0  in. 
with  respect  to  time  for  all  SHM  systems  found  in  the  field. 
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